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ABSTRACT 

Reviews of research in special education have a unique place in the scien- 
tific literature, with the potential to inform a broad audience about the 
effectiveness and acceptability of assessment and intervention strategies for 
a heterogeneous group of children and youth. Results from reviews are 
useful to school leaders, practitioners, parents, policymakers, grant funders, 
researchers, journal editors, and reviewers. Despite an international move- 
ment toward the implementation of evidence-based practices, which 
includes quality indicators for reviews of research, the field of special 
education has not yet embraced its own set of review indicators. In this 
manuscript, we make the case for and propose a set of quality indicators for 
special education, with the goal of making all reviews in the field trust- 
worthy, transparent, and systematic. Attaining a high level of quality in 
research reviews will then help researchers guide consumers to select the 
most effective interventions for youth with disabilities. 


Research in special education can appeal to a broad audience, including practitioners, parents, 
policymakers, grant funders, researchers, journal editors, and reviewers (Cooper & Hedges, 2009). 
Each of these constituents employs wisdom and experience, often through conversations with 
influential members of their social networks, to interpret findings from research and make sound 
decisions in selecting, implementing, and communicating effective assessment and intervention 
strategies (Daly, 2010; Kavale, 2001). Constituents benefit from rigorous reviews that assist them 
in becoming wiser, better consumers of research. It is the responsibility of special education 
researchers to combine and communicate the results of our work to this diverse group of constitu- 
ents. An essential first step in doing so is to conduct a broad range of clearly reported, objective, and 
transparent reviews of research with a set of quality indicators. Despite the emphasis in recent years 
on developing and disseminating discipline-specific quality indicators for a wide range of research 
designs (e.g., Moeller, Dattilo, & Rusch, 2015; Trainor & Graue, 2014), there remain none for 
systematic reviews, arguably the most important vehicle for describing and disseminating research 
findings in special education. 


Need for quality indicators in special education research 


There is an urgent need for researchers in special education to produce high-quality reviews of 
research. According to Bridgeland and Orszag (2013), former officials in the Bush and Obama 
administrations, respectively, a mere $1 of every $1000 spent by the federal government for 
interventions and programs, including those in education, is supported by research. Spending 
decisions, according to the authors, “are largely based on good intentions, inertia, hunches, partisan 
politics, and personal relationships” rather than evidence of effectiveness (para. 1). This practice not 
only wastes federal dollars, but also results in ineffective and potentially harmful programs and 


CONTACT Elizabeth Talbott, PhD o Etalbott@uic.edu © College of Education, Department of Special Education, University of 
Illinois at Chicago, 1040 West Harrison St. M/C 147, Chicago, IL 60607-7133, USA. 


© 2018 Taylor & Francis 


246 (©) E. TALBOTT ET AL. 


interventions delivered to youth with disabilities. We need only examine the proliferation of 
treatments marketed to parents of children with disabilities—without a shred of evidence, or with 
previous evidence of ineffectiveness and harm, to realize the vital contribution special education 
researchers can make in preparing high quality systematic reviews (e.g., Schreibman, 2005). 

For example, over the years, dietary interventions have been proposed to change behavior for 
students with disabilities, and systematic reviews of research have uncovered their lack of effective- 
ness. Examples of those interventions include (a) the Feingold diet designed to reduce hyperactivity 
(Kavale & Forness, 1983) and (b) gluten- and casein-free diets to reduce aggression and promote 
communication, social interaction, and motor and cognitive abilities for students with autism 
spectrum disorder (Zhang, Mayton, & Wheeler, 2013). In addition, systematic reviews of research 
have revealed potentially harmful contexts for the delivery of otherwise effective interventions. For 
example, in efforts to reduce aggressive, rule-breaking, and delinquent behavior, researchers have 
found that grouping aggressive youth together to deliver those interventions can be harmful; that is, 
when effective interventions are delivered to groups of aggressive youth, as opposed to groups of 
aggressive and nonaggressive youth together, these interventions are likely to promote, rather than 
reduce aggressive behavior (Dodge, Dishion, & Lansford, 2006; Petrosino, Buehler, & Turpin- 
Petrosino, 2013). 

The international movement in special education and related fields (e.g., psychology, criminal 
justice, social policy, and health care) toward the implementation of evidence-based practices 
provides the incentive to researchers to improve the quality of their research reviews. Indeed, one 
could argue that systematic reviews of research are at the center of the evidence-based practices 
movement (Cook & Odom, 2013; Shavelson & Towne, 2002; White, 2009). The need for quality 
indicators for research reviews is particularly imperative in special education, given (a) the extra- 
ordinary heterogeneity in the population of individuals served; (b) the diverse settings in which 
individuals with disabilities are educated; (c) the rich cultural and linguistic diversity of the popula- 
tion; (d) the heterogeneity of research designs used by researchers in the field; and (e) the diverse 
consumers of research associated with special education (Odom et al., 2005). Because individuals 
with disabilities represent some of the most unique students who attend public schools, systematic 
reviews of assessment and intervention strategies serving them need to be as high quality as possible. 
How do consumers make sense of findings from diverse research in special education? How do 
special education practitioners and school leaders make clinical decisions for such a heterogeneous 
population of youth? They do so by exploring the results of high quality research reviews addressing 
the unique needs of youth with disabilities (Jefferson, Wager, & Davidoff, 2002). 

Researchers in special education have established quality indicators for studies employing four 
types of research designs: experimental and quasi-experimental designs, single subject designs, 
correlational designs, and qualitative designs (Odom et al., 2005). Across the different types of 
research design, these indicators address the need to clearly conceptualize the research study, identify 
study participants and settings, and describe and specify relevant research procedures to increase the 
reliability of the findings, as well as data analysis techniques. Further, as research syntheses in the 
social sciences proliferate, and in areas of study with a substantial number of reviews such as peer 
tutoring, researchers will develop more overviews, or reviews of systematic reviews 
(Polanin, Maynard, & Dell, 2016). The quality of these overviews will depend on the quality of 
systematic reviews on which they are based (Polanin et al., 2016). Clearly, the time is right for the 
field of special education to have its own set of quality indicators for research reviews; the current 
study provides the first step in the process of establishing and testing these indicators. 

In this article we reveal that systematic reviews are highly flexible and not inextricably tied to studies 
using a particular type of research design. Instead, they can be employed to address a variety of 
research questions with studies employing a wide range of designs (Gough, Oliver S, & Thomas 2013). 
The key to a systematic review is transparency and objectivity in the methods and comprehensive 
search procedures, as well as attention to methods that can clearly affect the outcome of the review 
(e.g., such as the search for unpublished studies and the evaluation of study quality). Consistent 


EXCEPTIONALITY (S2) 247 


application of these indicators will help to reduce bias in reviewing and enhance the believability and 
validity of the results. With results from a body of transparent and trustworthy reviews, researchers can 
and should work to communicate the direct applications of their research findings with stakeholders. 

For example, reviews of research can be used by school leaders and practitioners to select 
assessment and intervention methods; by parents to explore the effectiveness and appropriateness 
of different interventions; by policymakers to weigh the costs and benefits of adopting particular 
strategies; by researchers to identify the strengths and weaknesses within an area to assist with 
planning their next study; by expert review panels to develop guidelines and practice standards; by 
grant funders to determine topics that are in need of additional attention and resources; and by 
editors and peer reviewers to judge the merits of manuscripts submitted for publication. It is the 
responsibility of researchers to conduct high quality reviews. The next step is to develop strategies to 
share the results of this work. 


Types of research reviews 


Approaches to reviews of research can be categorized in the following ways: as narrative literature 
reviews, systematic literature reviews, quantitative literature reviews, and meta-analyses 
(see the Campbell and Cochrane Collaborations; Cooper et al., 2009; Gough et al., 2013). Across 
all types of reviews, researchers are guided by research question(s), selecting studies using research 
designs common in special education: experimental and quasi-experimental group designs; single- 
case designs; correlational designs; and qualitative designs (Odom et al., 2005). Table 1 provides a 
summary of the characteristics of the types of reviews. Here we provide definitions from different 
sources, along with distinctions among them. 

Narrative reviews are written on a specific topic, and can include studies with diverse research 
designs. In fact, a particular strength of narrative literature reviews is their methodological diversity: 
in a narrative review, researchers are able to combine evidence across different types of research 
designs to address their research questions (Baumeister & Leary, 1997). In their narrative reviews, 
researchers can formulate a new theory based on extant research, test hypotheses, or examine the 
evidence underlying assessment and intervention practices. However, narrative reviews are not 
always conducted with consistency and transparency, and often contain a review of select studies, 
rather than of all studies available on a given topic (Lane, 2009). As is the case with narrative reviews, 
systematic reviews are conducted to address a specified topic, but provide a formal and transparent 
description of research methods that are open to public inspection, with a goal of including all 
studies that meet the inclusion criteria (Cooper et al., 2009). Quantitative reviews and meta-analyses 
are particular types of systematic reviews. A strength of systematic and quantitative reviews and 
meta-analyses is their focus on clear and transparent methods; however, meta-analyses do require a 
focus on a single type of research design within a given review (e.g., experimental and quasi- 
experimental or single-case research designs). 

Just as there is a need for a variety of research designs, each type of review has its place in the field of 
special education and can have many practical purposes. Reviews vary in the depth and breadth of the 
topics they cover, and in the type of research questions they address (Gough et al., 2013). Systematic 
reviews are likely to be of the most help to school leaders as they select evidence-based interventions for 
their students, as required by the Elementary and Secondary Education Act (Sparks, 2016). For example, 
in a recent study of three state education agencies and their strategies for selecting research-based 
interventions for low-performing schools, professionals identified reviews of research as a frequent 
source of information for effective interventions (Goertz, Barnes, Massell, Fink, & Frances, 2013). By 
improving the overall quality of research methods applied to reviews of research, special education 
researchers provide a critical service to these stakeholders who seek effective activities, strategies, and 
interventions for their students, in compliance with federal law. 
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Table 1. Systematic revievv quality indicators. 
Quality Indicator 


Provide a conceptual context for the research and state clear 


research questions and objectives. 


Develop and report eligibility criteria 


Description 


Define the problem in the context of relevant theory and 


research. This includes (a) summarizing previous research, 
including the importance of the review to the field; (b) 
defining the key variables and phenomena under 
investigation so that relevant and irrelevant studies can be 
identified; (c) clearly stating the purpose of the review which 
might be to formulate a new theory based on extant 
research, to test hypotheses, or to examine the evidence 
underlying assessment and intervention practices; (d) 
indicating the types of participants who will be included in 
the studies to be reviewed and citing relevant research in the 
introduction; (e) noting the type of research that will be 
reviewed and providing an appropriate rationale for the use 
of the paradigm in relation to the topic; (f) providing a 
clearly stated research question to guide the review. 


Clearly describe criteria for inclusion and exclusion of studies 


including (a) the operational features of the independent 
variable, dependent variable, or educational phenomena under 
study; (b) the research participants eligible for inclusion; (c) the 
research designs and other methodological features used to 
exclude studies; (d) the time period from which the studies 
could be drawn; (e) whether unpublished literature such as 
dissertations or conference proceedings were eligible; and (e) 
any language, cultural, or geographic restrictions. 


Report search procedures used to identify potentially eligible Provide a comprehensive description of the search procedures 


research studies. 


including (a) the explicit identification of databases searched, 
including databases of unpublished studies; (b) the keywords 
used within these databases and other delimiters such as 
publication year; (c) the additional procedures used to identify 
eligible research studies such as an examination of the citation 
lists of relevant review articles; the use of hand search procedures 
in relevant journals; the review of the citation list by subject 
matter experts to ensure all relevant studies have been found; 
the review of citation lists from included studies; (d) review of 
titles and abstracts (or similar process); and (e) the number and 
qualifications of the individuals conducting the initial search. 


Report procedures for retrieving studies and document the Describe the procedures used to retrieve studies and the 


proportion of potentially eligible studies retrieved. 


methods for cataloguing the studies. This includes reporting 
(a) the number of unique studies identified through the 
various data-based and non-data-based searches; (b) the 
number of studies excluded after the initial screening of 
titles and abstracts; and (c) the number and proportion of 
studies identified as potentially eligible that were 
successfully retrieved for full review. 


Describe methods used to screen studies for inclusion and Describe the procedures for screening studies for eligibility 


document reasons particular studies were not included. 


including (a) the number of studies successfully retrieved 
and subsequently screened out because they did not meet 
eligibility criteria; (b) the reasons the excluded studies were 
deemed ineligible; (c) the total number of studies that met 
eligibility criteria and will subsequently be included in the 
review; (d) the training and expertise of those charged with 
conducting the screening process; (e) reliability or 
interobserver agreement statistics used to evaluate the 
consistency of the screening process; and (f) the method 
used to resolve any disagreements amongst screeners. 


Develop a coding scheme that addresses critical aspects of the Clearly state the procedures used to develop the coding scheme 


research. 


with reference to the relation item categories for addressing the 
questions and objectives of the review. The general procedures 
for the following category should address (a) the expertise and 
training provided to individuals charged with coding studies; 
(b) the number and proportion of studies that were coded by 
more than one independent observer; (c) the reliability or 
interobserver agreement statistics used to evaluate the 
consistency of each domain of the coding scheme; (d) the 
procedures used to resolve disagreements amongst coders; (e) 
the response categories available for coders to select from; and 
(f) the procedures used to address issues for missing 
information within research studies. 


(Continued ) 
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Table 1. (Continued). 


Quality Indicator Description 


Describe adherence to key indicators of study quality. Evaluate the quality of studies included in the review to 
determine the extent to which conclusions drawn are based 
on research standards of the particular area. The specific 
indicators used will be based on the type of research 
sampled and the discretion and expertise of the researchers 
conducting the review. 

Describe the research participants and contexts included Conduct a descriptive review of the relevant research 

across the studies. participants to assist consumers determine which 
populations and conditions the results apply. Items relating 
to students might include (a) gender, (b) grade level, (c) 
diagnostic information, (d) disability status, (e) 
socioeconomic status, (f) primary language, and (g) setting 
information. The research team should consider other 
conceptually relevant sample information. 


Describe the variables under study to evaluate the Provide a description of the variables under study to assist the 
consistency of operational descriptions and data collection consumer determine the interventions, outcomes, and other 
methods used. factors under study to which the research might generalize. 


Items relating to variables might include (a) how the variable 
was operationalized, (b) the methods used to collect the 
variables, and (c) the quality of the system used to measure the 
variables under study. 

Develop a data analysis plan to address the research Describe the procedures used to analyze the data to address the 
questions. research questions with the criteria for explaining patterns of the 

systematic coding and describing variability in results across 
studies evident. These procedures will be based on the type of 
research examined and the particular information obtained. This 
will include considering (a) the method for aggregating the 
results of the application of the coding scheme to describe 
patterns within the literature for the particular areas being 
investigated; (b) the specific set of methods used to determine if 
an intervention or assessment is empirically supported if 
applicable; (c) the procedures used to determine the contexts, 
settings, and individuals for whom a particular practice or set of 
findings is applicable; (d) the process of synthesizing study results 
to identify meaningful patterns; and (e) whether statistics were 
used to describe the research findings, the type of information 
these statistics provide, and the reasons these methods were 
employed to address the research questions. 

Interpret the results considering the research findings, study Interpret the research findings within the context of the 
quality, and conditions to which the research might methodological rigor of the systematic review, the evidence 
generalize. derived from the coding schemes, and the analysis of 

research findings. The interpretation of the results entails (a) 
characterizing the primary objectives of the research; (b) 
describing the results of the review within the context of the 
objectives; (c) providing comment on the methodological 
strengths and weaknesses of the literature and how these 
should be considered within the context of the results; (d) 
describing the generalizability of the conclusions including 
the relevant student and teacher populations as well as the 
appropriate contexts and variables; (e) acknowledgement of 
the limitations of the current review; (f) providing 
implications of the review for relevant domains such as 
research, practice, policy, and theory as applicable. 


Note. Quality indicators are derived from the following sources: Campbell and Cochrane Collaborations (www.campbellcollabora 
tion.org; www.cochrane.org); Cooper et al., 2009; What Works Clearinghouse (US Department of Education, 2002); Sackett et al. 
(2000); University of London, Institute of Education (2010). 


Publication of research reviews in special education 


Over the past quarter century in special education, researchers have seen the rise of reviews. 
Figure 1 reveals that the co-occurrence of the term “special education” with “systematic review” 
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Figure 1. History of systematic reviews in special education. 


Note. The number of returns per year from Google Scholar using the keywords “special education” with “systematic 
review” or “meta-analysis” 


and/or “meta-analysis” within Google Scholar has increased steadily in the past 25 years. 
Specifically, these terms occurred together approximately 250 times in 1990 and more than 
7000 times in 2015. This pattern roughly mirrors patterns of increase in reviews of health care 
interventions. In the past 20 years in health care, research reviews have increased seven-fold, 
from approximately 500 per year in 1993 to 3500 per year in 2005 (White, 2009). As such, there 
has been a clear upsurge in interest for systematic reviews in special education over the last 
quarter century. 

Herein we present a set of quality indicators for the methods of research reviews in special 
education (see Table 2). The purpose of our work is to provide guidance for improving the quality of 
reviews in our field, with the ultimate goal of improving the lives of children with disabilities and 
their families. This article focuses on methods for investigators to improve the conduct and 
presentation of reviews. We argue for establishing quality indicators for research reviews that can 


Table 2. Definitions of the types of reviews common in special education. 


Narrative literature Review of research on a specified topic, using highly flexible methods. Can include studies with diverse 
review research designs. Can be written to advance theory in a field, based on extant research. 

Systematic literature Research review on a specified topic that provides a formal and transparent description of the 
review rationale for inclusion of studies, as well as the method for searching the literature to identify 


studies; the criteria for determining eligibility; and the method for screening, assessment, and 
coding or summarizing information from studies. Systematic reviews use explicit methods to 
identify, select, and critically appraise relevant research, and extract and analyze data (as 
appropriate) from studies. 


Quantitative literature Review meets all features of a systematic review. Estimates quantitative values from included studies; 


review examines patterns in the results across studies without deriving a single, summary statistic. Provides 
the opportunity to conduct an informal review of the estimates derived. 
Meta-analysis Review meets all features of a systematic review and also aggregates estimates from individual studies 


into a single, summary statistic for the body of research. Provides the opportunity to conduct a 
formal statistical analysis of the heterogeneity around the point estimate to identify underlying 
reasons for the variability in findings across studies. 


Note. The definitions in this table are adapted from those employed by Cooper et al., 2009; the Campbell and Cochrane 
Collaborations, www.campbellcollaboration.org; http://www.cochrane.org/; What Works Clearinghouse (US Department of 
Education, 2002); and the University of London, Institute of Education (2010). 
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be applied across the different types of reviews. Through transparency and improved quality in the 
methods of reviewing, researchers can identify the extent to which a subsequent review represents a 
replication, or a new study. That is, with sufficient detail provided in the methods, readers can judge 
the specific contribution of the research for themselves. 

We argue that researchers in special education should conduct their reviews with scientific rigor: 
providing detailed description of their methods and transparency about each decision they make, 
including assessment of study quality (Lane, 2009). Each review should address why it has the focus 
it does, rather than attempting to address all issues for every reader (Gough et al., 2013; University of 
London Institute of Education, 2010). This level of rigor will help contribute to the goal of improving 
the quality of research in special education, so that researchers can communicate research findings to 
constituents. 


Quality indicators for systematic reviews in special education 


The following section provides an overview of the quality indicators that we recommend for 
researchers who conduct reviews on the variety of questions important for moving the field forward 
regardless of research paradigm or review purpose. As mentioned, we have derived these indicators 
from several sources, including the highly-cited handbook of research synthesis edited by Cooper 
et al. (2009); criteria established within the area of evidence-based medicine (Pai et al., 2004; Sackett 
et al., 2000); and the methods and strategies endorsed by key review organizations such as the 
Campbell and Cochrane Collaborations and the What Works Clearinghouse (US Department of 
Education, 2002). In doing so, we have sought to select those indicators across fields that appear to 
be applicable to special education research reviews. 

There is no gold standard in systematic reviewing, and no guarantee that quality indicators will 
result in unbiased reviews. However, by applying quality indicators and transparency in methods, 
researchers can reduce bias. Researchers who conduct different systematic reviews and meta-analyses 
on the same topic may obtain disparate findings as a result of different research questions, data 
sources, search strategies, timing of search and retrieval, inclusion and exclusion criteria, outcomes, 
populations, and study designs selected, as well as errors in primary data and/or data extraction, 
along with varying methods used for data analysis (Ioannidis, 2010). We argue that the application of 
a consistent set of quality indicators in special education research reviews is a vital first step to 
reducing bias, and to assisting diverse consumers of research. Maintaining transparency in methods 
is a first step in insulating the science of systematic reviews from political or ideological bias 
(Cordray & Morphy, 2009). 


Developing a research question 


The initiation of a systematic review requires the development of a research question to assist the 
investigator in articulating the overarching goals of the review (Cooper & Hedges, 2009). Research 
questions may be presented in the context of a theoretical framework and prior research conducted 
on the topic; well-designed and testable research questions lead to the development of clear 
objectives and provide boundaries for making decisions about which studies to include in the 
systematic review. Research questions can range from determining the effectiveness of an interven- 
tion, to identifying common patterns or themes across qualitative research on a given topic, to 
evaluating the strength and generalizability of a correlational claim or assessment method 
(Nye, Melendez-Torres, & Bonell, 2016). 

The level of detail included within research questions will depend on the particular topic and the 
intended breadth of the review. However, there are common components of the research question 
regardless of the content or methodological areas being sampled. Specifically, the questions should 
identify the population of participants, the primary variables, and the outcomes of interest 
(Cochrane Collaboration, 2013). For example, if the purpose of the review is to determine whether 
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a particular program has sufficient evidence to warrant its use in schools for reducing student 
aggression, research questions might include information pertaining to the grade levels and disability 
status of students; the specific name or broad class of the intervention; and the specific outcomes of 
interest such as disruptive behavior or teacher ratings of student conduct. We provide sample 
research questions for various systematic review topics in Table 3. Information included in the 
research questions can be expanded to incorporate particular methods, time frames, and cultural or 
linguistic diversity of participants, all of which may be pertinent to the goals of the review. When 
researchers develop questions containing additional information, they have greater guidance in 
developing eligibility criteria for included studies. 


Developing eligibility criteria 


The development of a research question is a vital first step in conducting a systematic review; questions 
guide researchers in communicating the overall purpose of the review and lead the researcher to 
identify research objectives. However, a clear research question alone is not sufficient for initiating the 
search and subsequent retrieval of studies. The primary purpose of providing eligibility criteria is to 
ensure that the decisions to include or exclude studies are clear so that readers have the opportunity to 
judge the applicability and limitations of the research (Wilson, 2009). 


Table 3. Sample research questions for various systematic review topics. 


Systematic 

Review Topic General Form Examples 

Qualitative What are the views of particular individuals on a given What are the perceptions of general education 
Themes and educational phenomena? teachers of including students with 
Patterns intellectual disabilities in science classes? 

How do students without disabilities view their 
classmates with disabilities? 

What do school personnel consider to be the 
key barriers to successful and sustained 
implementation of effective practices? 

Documenting What is the extent of the association between two or more Do students with higher anxiety tend to have 
Correlational variables or outcomes? lower achievement? 
Evidence How likely are students with learning 
disabilities to obtain a post-secondary 
degree? 


How do the rates of postsecondary transition 
for students with disabilities differ based on 
the family income? 


Evidence-Based Does a particular assessment have demonstrated reliability, How do special and general education teachers 
Assessment validity, and diagnostic accuracy across various study differ in their ratings of student externalizing 
samples? and internalizing behavior? 

Do curriculum based measures predict reading 
achievement on standardized tests? 

How much association do the results of 
transition planning assessments demonstrate 
with postsecondary outcomes for students 
with disabilities? 

Evidence-Based Does a particular intervention result in improved academic Do group contingency interventions result in 
Intervention or behavioral performance for students? improved behavior for students with 
emotional and behavioral disorders? 


Does performance feedback lead to increased 
rates of teacher fidelity to academic 
interventions? 

Does vocabulary instruction help students with 
cognitive impairments to improve their 
reading achievement? 
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Because the hallmark of a systematic review is replicable research procedures that strengthen 
study conclusions and allow readers to evaluate those conclusions, the researcher needs to provide 
explicit detail regarding inclusion and exclusion criteria and the search process designed to answer 
the research question. These eligibility criteria are designed to build on the research question and 
objectives, with the goal of helping the researcher make transparent decisions about which studies to 
include and exclude. 

Eligibility criteria applied to each review will vary depending on the particular purpose and goal 
of the research, but certain information does apply to the majority of reviews (Davies, 2000), 
including (a) specific demographic, behavioral, and academic information that distinguishes the 
student population; (b) the particular contexts in which the research has taken place; (c) the 
constructs and outcomes being reviewed and whether there are any restrictions on how these were 
measured; and (d) the research methods that are appropriate, along with any critical distinctions, 
such as including only those studies that randomly assigned participants to intervention or control 
conditions or used a particular sampling procedure within qualitative research designs. The elig- 
ibility criteria described here are not exhaustive, but are meant to provide general guidance on some 
of the more typically used criteria. 


Identifying eligible research studies 


Following the development of a research question and accompanying eligibility criteria which 
provide the basis for determining which studies will be included in the review, the challenge then 
becomes finding the population of studies that meet the eligibility criteria (Sampson et al., 2009). 
The emphasis is on identifying all eligible studies on the topic using transparent research methods, 
so that the findings of the review are as objective and replicable as possible. That is, the results of the 
review depend directly on the findings from original research and, therefore, any missing informa- 
tion can affect the reviewers’ conclusions. The rigor and transparency of the review is a direct result 
of the researcher’s identification and collection of all published and, to the extent possible, unpub- 
lished research (Papaioannou et al., 2010). Therefore, researchers must describe procedures for 
searching the literature. Procedures include identifying the bibliographic databases searched, along 
with explicit descriptions of keywords used within these databases. This process allows readers to 
determine the likelihood that all relevant studies have been found, and to be able to replicate the 
search. 

Locating a high proportion of eligible studies will require conducting the search in several 
different databases, with careful consideration of the range of terms used across these databases, 
often from different disciplines. It is essential at this stage for researchers to include a comprehensive 
search for unpublished studies, such as dissertations. Indeed, Pigott and colleagues (2013) found that 
all 96 research universities in the United States with very high research activity, according to the 
Carnegie classification (McCormick & Zhao, 2005), had access to the ProQuest dissertation and 
theses database. From there, researchers can locate dissertations online or through interlibrary loan. 
Moreover, the researcher may take additional steps to ensure that all eligible studies have been 
identified, such as conducting hand searches of specific journals, examining the reference lists 
contained within related review articles and eligible studies, and contacting experts on the particular 
topic to ensure the list is comprehensive. The reporting of these steps is critical for the reader to 
determine whether the review includes the full population of studies on the topic and whether 
studies with common features have been overlooked. The intentional or unintentional exclusion of 
relevant studies raises the possibility that the review findings are not representative of the broader 
population of research findings, or that the methods introduce bias. In addition, explicit reporting of 
the steps at this stage allows future researchers to replicate the process. 
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Retrieving eligible research studies 


To identify eligible studies, researchers must first conduct a search of databases and previous 
reviews, reviewing each title and/or abstract to develop an initial pool of studies for further review 
(Papaioannou et al., 2010). At this point in the process, the goal is for researchers to attempt to 
retrieve original articles for collection in the database, and review these for possible inclusion. The 
strategies for retrieving and storing articles are beyond the scope of the article; however, it is likely 
that reviewers will not be able to collect all studies listed. Thus, researchers must record the studies 
not captured. This record provides readers with vital information to evaluate the rigor and believ- 
ability of the research. That is, the creditability of the review is compromised when only the easy-to- 
find studies are included (Gough et al., 2013). Reviews should contain the fewest possible omitted 
studies, allowing readers to determine the extent to which results are reflective of the broader 
population of research findings available. 


Developing a coding scheme 


Research questions associated with narrative reviews may or may not necessitate study coding by the 
authors. However, coding information from original research can be enormously helpful to researchers 
who seek to identify patterns and inconsistencies in results across studies (Wilson, 2009). To enable 
study coding, researchers develop a protocol that guides them in obtaining information from eligible 
studies and recording it in a format that can be readily analyzed. Because coding schemes are 
developed to address the research questions of a given study, the particulars of each protocol will 
inevitably differ. Despite this inherent variability, specific coding categories can be employed in a 
majority of reviews, given their importance for evaluating both the rigor and generality of research 
drawn from various paradigms. We provide examples of these categories in Table 4 and describe their 
utility next. 


Assessing study quality 

Study quality refers broadly to the degree to which the research design procedures generate valid 
results (Valentine, 2009). Authors of high-quality systematic reviews are dependent on clear, 
transparent, operational descriptions of intervention procedures in the original research in order 
to identify those intervention procedures that are most effective for particular individuals with 
disabilities (Gersten et al., 2005; Maggin et al., 2014). This level of detail is certainly a measure of 
study quality; in addition, systematic reviewers can screen for additional measures of study quality in 
the articles they review for inclusion, and/or code the quality of the work that has been included. It 
behooves reviewers to adopt clear guidelines for assessing study quality across all components of the 
study, but perhaps particularly with regard to descriptions of intervention. 

That is, systematic reviewers need to make decisions and apply judgments about study quality 
consistently to each of the studies included in their reviews—either at the screening stage, or the 
coding stage, or both stages (Lane, 2009). Study quality is indicated by the degree and precision of 
information presented throughout the original research. 

Indeed, study quality is multidimensional and can be assessed during multiple stages of the 
research review (Valentine, 2013). That is, in conducting reviews, researchers may decide in advance 
to include only those studies using group experimental designs employing random assignment 
(Valentine, 2009); this would be an initial measure of study quality. Once studies meet an initial 
criteria for inclusion, researchers can further assess quality, depending on the goals of a review. 
Elements of study quality to be assessed include threats to internal validity (e.g., participant attrition 
or participant crossover from control to experimental group), construct validity (e.g., the adequacy 
of labels applied to constructs in a given study), and external validity (e.g., the extent to which study 
results be generalized). 
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Table 4. Example categories and items to be included in coding schemes of systematic 


reviews. 
Coding Domain Example Items 
Source Information ° Study identification number 
Report identification number 
Citation information 
Study Quality ” Research design 
Data sources 
Sample size 
Reported attrition 
Other methodology specific criteria 
Participant Characteristics © Gender 
Age 
Grade 


Disability status 

Socioeconomic status 

Primary language 

Achievement level 

Diagnostic information 

Setting Characteristics " Classroom type 

Type of instructional activity 

Teacher characteristics 

Independent variable " ndependent variable description 
Measurement of independent variable 
Reliability of independent variable 
Components of independent variable 
Fidelity of independent variable (if applicable) 

Dependent variable © Outcome variable of interest 
Definition of outcome variable 
Method of data collection 
Unit of measurement 
Time points of outcome 
Reliability of data 


Note. ° These items should be included for each individual report for organizational 
purposes; ” These items will be based on the particular designs being included and will 
require reviewer methodological expertise, “ These items will be based on the particular 
populations, context, and variables being investigated. 


Study quality assessments can include those associated with the research design of a particular 
study. Examples include (a) the number of phase comparisons used to determine whether experi- 
mental control has been established within a single-case research study; (b) whether an acceptable 
level of reliability for a measurement system in single-case design has been reported; (c) the extent to 
which triangulation of data, investigators, and methods are evident in qualitative studies; (d) whether 
a qualitative research team took steps necessary to account for possible reflexivity that could 
influence interpretations; and (e) the reporting of score reliability coefficients and their correspond- 
ing confidence intervals for research in the psychometric defensibility of particular measures. 

These are a few examples of ways in which researchers might assess study quality. Clearly, the 
assessment of study quality within a given review must be linked to the goals of that review. 
Researchers have developed study quality rating scales for reviewers to use; however, these scales 
vary widely, and review results can differ substantially depending on the quality of the scale one has 
chosen (Valentine, 2013). Therefore, reviewers should evaluate the quality of study methods using 
multiple techniques that reflect sensitivity to the individual studies being reviewed, in the context of 
the goals for the systematic review. 


Describing participants and settings 

The evaluation of study quality is important for establishing confidence in the research results; 
however, even the results of the best-designed studies require consideration of the individuals and 
contexts to whom the results can generalize (Valentine, 2009). Systematic reviewers are well 
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positioned to provide a detailed overvievv of the characteristics of participants and settings 
addressed in a particular body of vvork. For example, a research team conducting a revievv on 
the effects of a particular sensory device at reducing stereotypy among students vvith autism 
might consider a range of student characteristics and/or settings in which the research has been 
conducted. Researchers can then ascertain the extent to which interventions work for particular 
students in particular settings, and whether gaps exist that can be addressed through future 
research. Researchers conducting systematic reviews can summarize and discuss student and 
contextual characteristics to provide guidance regarding the conditions to which the findings 
apply, and where the gaps are. 


Describing the variables under study 

Well-conducted systematic reviews increase one’s confidence in the findings and subsequent 
conclusions over less methodical approaches to summarizing research because of the transparency 
of research methods. The application of a systematic approach alone, however, does not make 
these types of literature reviews interesting, enlightening, or informative. Systematic reviews are 
compelling because they organize information across several sources to draw conclusions that are 
more generalizable than those from a single study. This includes the diversity inherent across 
research participants and contexts included in different studies as well as the variables and 
phenomena being investigated. As a result, those engaged in conducting systematic reviews should 
aim to describe the diversity in the variables used across studies, including important moderator 
variables. Examples of important moderator variables for systematic reviews of interventions 
include fidelity of intervention implementation; duration, frequency, and intensity of intervention; 
and intervention cost (Cordray & Morphy, 2009). 

For example, a systematic review of a particular reading program might examine variability in the 
procedures used to implement the intervention. This might include developing codes regarding 
whether the interventionists were graduate research assistants or teachers; details regarding the 
training provided to implementers; or the amount and length of intervention sessions. Similarly, a 
qualitative meta-synthesis might consider variability regarding themes and data sources across 
studies. In contrast, a qualitative meta-synthesis might consider variability with regard to themes 
and data sources across studies. For example, Scruggs, Mastropieri, and McDuffie (2007) investigated 
the perceptions of school personnel engaged in co-teaching in inclusive settings. Their research team 
used a recursive coding process in which the procedures were continually refined over the course of 
the research. They applied an inductive process of data analysis to assist them in identifying relevant 
patterns and themes. Narrative reviewers can summarize key variables across studies in a series of 
tables. Researchers and systematic reviewers must provide a precise, operationalized description of 
intervention and outcome measures, so as to facilitate replication and future systematic reviews 
(Gersten et al., 2005). 

Examining variables across studies allows reviewers to ascertain what constructs researchers are 
studying. Looking across studies, reviewers can determine the uniformity of construct definitions. For 
example, reviewers might conduct a systematic investigation of variability in operational definitions 
and data collection procedures for the independent variables, dependent variables, and educational 
phenomena under study. Such variability will ultimately depend on key factors such as research 
questions, topic of study, and study measures. It is the reviewer’s responsibility to develop an approach 
for capturing this variability in a systematic way, so as to describe it to readers of the review. 


Extracting research findings from eligible studies 


The purpose of a systematic review is to provide a methodical and transparent approach for 
synthesizing research to address a well-defined research question (Cooper & Hedges, 2009). The 
application of the coding scheme to eligible research studies provides information that can be used 
to assist with interpreting the results obtained across the included studies. However, the research 
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question is ultimately addressed through the collection and subsequent synthesis of the results reported 
across retrieved reports. Therefore, it is important for the reviewer to provide a description of the 
procedures used to extract the research findings from each study (Cochrane Collaboration, 2011). The 
specific methods employed will relate to the particulars of the research question and the specific 
research designs eligible for inclusion. For instance, a review of single-case research should provide 
detailed information on the approaches used to estimate the values of graphed data points; whereas, an 
analysis of experimental and quasi-experimental group-based research would need to report the 
procedures for extracting information relevant for computing effect sizes. Similarly, a meta-synthesis 
of qualitative research might include a description of the procedures used to identify the relevant 
themes defined within each study. Narrative reviewers will need to be explicit regarding what 
information was extracted from each study, with pertinent information displayed from each study, 
ideally in tables. 


Developing a data analysis plan 


The goal of a particular plan for data analysis is to synthesize the results from the review; synthesis is the 
process of integrating findings from the studies to answer the research question (Gough et al., 2013). The 
data analysis plan goes hand in hand with the research question. Following the extraction of data, the 
investigator is ready to synthesize the results obtained across studies. The specifics of the data analysis 
plan will depend on the type of data extracted and the overall goals of the systematic review. For example, 
quantitative data can either be statistically combined using meta-analytic procedures or summarized in 
table form, reported separately for each study. The advantage of meta-analysis relates to the improved 
precision of the statistical estimates obtained from aggregating quantitative results across samples drawn 
from several studies. These procedures also allow investigators the opportunity to estimate the consis- 
tency of the estimates through heterogeneity analyses and determine whether and to what extent 
conceptually relevant variables explain the variability in statistical estimates (Cooper et al., 2009). 

A full review of meta-analysis is beyond the scope of the present article and those researchers 
interested in posing systematic review questions that have a meta-analytic component are encouraged 
to consult various sources (e.g., Borenstein, Hedges, Higgins, & Rothstein, 2009; Cooper et al., 2009) 
and work alongside experienced statisticians and methodologists in the area. Moreover, it is important 
to note that the use of meta-analytic techniques does not increase the validity of the systematic review. 
Rather, these methods are useful for obtaining more refined quantitative estimates, describing the 
consistency of these estimates, and examining reasons why these differences exist across studies 
investigating a common topic. Regardless of whether the systematic review entails the use of statistics 
or narrative data such as that typically found in qualitative research and narrative reviews, the 
procedures for synthesizing the data to identify patterns across studies must be described in a 
replicable fashion to ensure transparency and provide readers with an opportunity to determine the 
appropriateness of the analyses. 

Likewise, a full review of qualitative meta-synthesis is beyond the scope of the present paper. In a 
meta-synthesis, “data from multiple studies are continuously and iteratively compared and inte- 
grated to achieve a new understanding of an underlying phenomenon and construct overarching 
narratives or new theories” (Kramer et al., 2012, p. 764). Researchers should consult available 
resources to learn how to systematically integrate evidence from multiple qualitative studies on a 
given topic (e.g., Dixon-Woods et al., 2005). 


Interpreting the results of systematic reviews 


The procedure of a systematic review leads to the generation of a diverse set of information relating to 
the findings across studies and other contextual and methodological characteristics of the research. The 
process of interpreting results from these varied sources requires careful and explicit consideration of 
methodological trends within the literature and the participant, contextual, and variable specific 
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features collected through the application of the coding scheme (Wilson, 2009). Full consideration of 
these factors will often require the review team to address limitations, inconsistencies, or ambiguities 
within the research. For example, the research team might find that the rigor of the research is 
insufficient to make strong claims despite the presence of some common trends; there might be a 
dearth of research to address support certain claims; or the results might be found to vary in 
accordance with particular study, sample, or contextual features. Authors should then be prepared 
to describe the nuances found within the research to accurately portray the literature base. 


Application of the quality indicators 


We applied the quality indicators described in the preceding sections to a sample of 12 systematic 
reviews of peer tutoring interventions for students with disabilities. The purpose of this application 
was to illustrate the variability in the methods used across reviews. We describe the details of this 
application in a chapter by Talbott, Trzaska, and Zurheide (2017), with the exception of inter- 
observer agreement for coding of the quality indicators, which we describe here. Specifically, two of 
the study authors coded quality indicators from the 12 systematic reviews, with 100% of the reviews 
double coded. Coders had previously participated in training and coding for another project using 
the same quality indicators. That training consisted of (a) an introduction to the study and the 
coding protocol presented by the lead authors; (b) practice coding with a subset of reviews and meta- 
analyses outside of the study of peer tutoring; and (c) computation of agreement statistics until there 
was at least 90% agreement on all codes. Coders were then given studies to code independently to 
ensure acceptable levels of coder agreement. We obtained 100% agreement across all codes in the 
sample of 12 systematic reviews of peer tutoring. We refer interested readers to Appendix A, 
Table Al, which details the results of this application for each of the 12 systematic reviews and 
meta-analyses we reviewed. In the following paragraph, we provide a brief overview of the main 
findings of the application. 

Results of the application indicated that across these systematic reviews, we found that reviewers 
addressed some methodological domains more consistently than others. For instance, each of the reviews 
included a set of clear, testable research questions or study purpose(s). Moreover, we found that authors 
of each of the systematic reviews provided information on the criteria used for study inclusion and 
exclusion. For the remaining indicators, however, reporting was more variable. Examples of methodo- 
logical domains with less consistent adherence included the reporting of search, retrieval, coding, and 
data analysis procedures. In each of these methodological domains, there were some reviews that 
addressed none or a very few of the criteria while others provided information for each of the indicators. 
The variability in reporting across these systematic reviews is exemplified by the finding that the number 
of indicators met across these reviews ranged from 26% ( = 11) to 73% (n = 31). These findings suggest 
that the methods used to conduct systematic reviews in special education might vary considerably from 
one review to the next. As the field continues to emphasize the importance of research, it is imperative 
that systematic reviews adhere to rigorous procedures to ensure that researchers are disseminating the 
most current and valid information. 


Standards for quality indicators 


We have presented these indicators along with a rationale for including them as a first step in a 
constructive contribution to the field. The evolution of quality indicators for research in single-case 
design provides an example of how the development and application of such indicators can proceed. 
First, Horner and colleagues (2005) established quality indicators and thresholds for single-case 
research; next, Kratochwill and colleagues (2013) developed What Works Clearinghouse standards 
for these same indicators. In the context of the advancement of quality indicators, their application 
by researchers in the field proceeds more slowly. For example, Moeller et al. (2015) examined the 
Horner and colleagues’ indicators and thresholds in 10 systematic reviews of 120 special education 
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studies and concluded that “considerable research is needed to reach consensus on what constitutes a 
single-case research quality indicator” (p. 151). 

In the development of standards, it is important to note that some indicators may be more 
consequential than others. For example, it is critical that special education researchers understand 
that restricting their reviews to published studies may result in biased findings for the review. In 
their studies of the peer review process (using Cochrane methods of reviewing), Jefferson and 
colleagues found that surprisingly little is known about the effects of external peer review on the 
quality of published information (Jefferson et al., 2002; Jefferson, Wager, & Davidoff, 2002). Yet, the 
assumption that having undergone peer review is a proxy for methodological quality is likely to be a 
flawed one (Smith, 2006). 

Indeed, including unpublished research in one’s systematic review is a critical component of 
reducing outcome reporting bias (Pigott et al., 2013). Pigott and colleagues compared findings 
reported by researchers in unpublished education dissertations with those reported by the same 
researchers in the published versions of their dissertations, and found that the odds of a statistically 
significant finding from a dissertation appearing in the published version were 2.4 times greater than 
the odds of a nonsignificant finding appearing in the published version. Special education research- 
ers may not be aware of the risk of outcome reporting bias associated with published research, nor 
the influence of studies of wide-ranging quality on their reviews. Indeed, not all studies are of equal 
quality, and some contain flaws (Lane, 2009). 

As is the case in any research, systematic reviews are vulnerable to bias. Reviewers can minimize 
bias in their reviewing by employing systematic methods described here. By taking a transparent 
approach to looking for scientific evidence and clearly describing their methods (Gough et al., 2013), 
particularly those methods most vulnerable to bias in systematic reviewing (e.g., formulating ques- 
tions, searching published and unpublished literature, addressing study quality, performing analyses, 
and reporting results), reviewers improve the creditability and believability of their reviews 
(Cordray & Morphy, 2009). Further, systematic reviews not only inform us about what is known 
in a particular field of study, but what is not known, identifying those areas of research in need of 
further research. 


Research to practice: The role of systematic reviews 


Systematic reviews can be vital tools in an effort to connect research to practice. Yet, how do special 
education researchers go about influencing policy and practice? Strategies must match the particular 
audience(s) with whom researchers seek to communicate. For example, as we have mentioned, the 
audience(s) for systematic reviews can be diverse, and include school leaders, practitioners, policy- 
makers, parents, grant funders, researchers, journal editors, and reviewers. 

Researchers can and must connect with school leaders and practitioners regarding the outcomes 
from high-quality systematic reviews. A recent policy brief from the Consortium for Policy Research 
in Education revealed that members of state education agencies used their professional social 
networks and the input of experienced practitioners to help them interpret and make sense of 
educational research, particularly systematic reviews (Goertz et al., 2013). Researchers at the London 
Institute of Education (2010) have argued for involving constituents from the outset in developing 
research questions for systematic reviewing, so that they have a stake in the outcomes. Having said 
that, researchers must remain independent of constituents in the actual conduct of research, so that 
results can remain trustworthy and unbiased (Cordray & Morphy, 2009). Researchers and reviewers 
of research can no longer view themselves as disseminators of results from on high; instead, they 
must communicate the results of their work and engage with constituents (practitioners, state 
professionals, and external partners) so that the work will be relevant, and therefore more likely to 
be applied by policymakers and practitioners. 

Parents of children and youth with disabilities are eager to know the results of research that can 
help their children. Through organizations such as the National Center for Learning Disabilities 
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and Autism Speaks, researchers can reach parents who are already active participants in the 
education of their children with disabilities. Researchers in the field of early language development 
have demonstrated a creative approach to communicate the effectiveness of scientific research to 
all parents, not just those who are actively involved in advocacy, by sharing simple strategies for 
language development through experiential learning in grocery stores and at family events in New 
York City (Hirsh-Paske & Golinkoff, 2015). This type of activity requires its own source of 
funding, but is an extremely valuable endeavor, especially to reach parents of children and 
youth from low income families, and those for whom English is a second language. 

By communicating the results of reviews in a clear and systematic way, special education 
researchers can engage funders in the development of the next line of research, and in the funding 
of educational outreach to parents of children with disabilities to support effective discipline 
strategies, for example (see parenting strategies found in Barkley, 2013; Patterson & 
Forgatch, 2005). Parents of children and youth across the economic spectrum are often eager to 
access the best services and interventions for their children, but may not possess the strategies or 
resources for doing so (e.g., Atkins et al., 2006). Armed with findings from a clear set of well- 
conducted systematic reviews, and with funding from private foundations, researchers can provide a 
lifeline to parents, helping them to access the skills, strategies, and resources for parenting and 
advocating for their youth with disabilities. 

Journal editors and reviewers will likely welcome the adoption of quality indicators for 
reviews in special education. At present, they have one set of indicators with authority behind 
it: the meta-analysis reporting standards, MARS (American Psychological Association, 2010). 
Yet, as we have pointed out, those standards may not be consistently applied by special 
education researchers who conduct all types of reviews (see Appendix A, Table A1). Thus, 
with a clear set of indicators available to them, editors and reviewers will be able to establish 
clearer expectations and feedback to authors who submit reviews. Additional research is needed, 
however, to develop field-specific indicators and guidelines for special education. For instance, 
the standards presented here would benefit from further validation and consensus building 
exercises. One example might be to follow the lead of others in our field (see Cook et al., 
2015), and conduct a Delphi study with expert researchers in special education, particularly 
those who have conducted high quality systematic reviews, in order to ensure that the guidelines 
are methodologically and conceptually sound. 


Summary 


Research in special education serves many purposes for a diverse set of constituents. In addition, the 
diversity of students served in special education requires that researchers have diverse research tools 
(including systematic reviews) at their disposal. Reviews of research have become more frequent in 
special education over the past 25 years, with researchers expected to produce more systematic 
reviews and overviews of those reviews in the future (Polanin et al., 2016). These reviews are more 
valuable and useful to our constituents as they become more systematic, transparent, higher quality, 
and less biased. 

Special education researchers have much work to do to communicate the results of systematic 
reviews of research to eager consumers. The first step is to ensure that the research reviews are of 
high quality, as “bad science can lead to as much mischief as no science” (Schreibman, 2005, p. 4). A 
central question addressed by researchers in an upcoming special issue of Child Development is this: 
how can scientists share the outcomes of their research in ways that maintain scientific integrity, and 
also influence practice (Hirsh-Paske & Golinkoff, 2015)? Systematic reviews have a clear role in this 
process, but they must be conducted with the highest standards in order to be trustworthy and 
valuable, not only to researchers, but also to members of the public who are direct beneficiaries of 
well-conducted, unbiased research. 
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